Towards Automated Exploration

The primary goal of auto-explore is to to establish a codebase that reduces the effort to produce a reasonable first-pass exploratory data analysis for a variety of dataset types.

This Python library is a first attempt at automating the process of exploratory data analysis – at least as far as computation and visualization is concerned.

Critical thinking is not included.

Potential Benefits of Semi-automated EDA

• Faster time to insights & modeling
• Shorter exploratory data analysis turnaround
• Reliable processes that are vetted & improved over time
• No need to re-configure old code to new situations
• Supplies a base for more in-depth analysis

Inspirational & Instrumental Works

This work necessarily relied upon many excellent open source Python libraries. Some of the inspirations and instrumental tools have been:

• pandas-profiling: This work was ultimately not used in this library due to neglect of the project.
• featexp: A great tool for visualizing univariate analyses of a target; useful pre-ML. This library was integrated into this one for experimentation on extending the work, and is used in producing univariate plots and feature selections.
• matplotlib: The bare-bones go-to viz package for Python. This package was relied upon heavily to produce this library.
• seaborn: Abstracts away from matplotlib by performing statistical analysis alongside plot generation. This package is also relied upon heavily.
• pandas: The go-to data wrangling tool for Python. pandas’ pd.DataFrame object as well as many of the tseries capabilities are leveraged.
• sklearn: An indispensable machine learning library that every data scientist should be proficient in using. All the functionality desired from this package has not yet been integrated into auto-explore, but there are plans to leverage this package more in the future.

Linear Regression Analysis

Text Analysis

Time Series Analysis

Correlation Analysis

Correlation Analysis

Categorical Analysis

Clustering Analysis

Clustering Analysis

Clustering Analysis

Helper Functions

In [10]: from auto_explore.datetime import make_calendars
In [11]: year_list = np.arange(2011, 2020)
In [12]: cal_df = make_calendars(year_list, drop_index=False)
In [13]: cal_df.head()                                                                                                          
Out[13]: 
              month  year    weekday  is_weekday  is_holiday  is_holiday_week
DATE                                                                         
2011-01-01  January  2011   Saturday           0           0                0
2011-01-02  January  2011     Sunday           0           0                0
2011-01-03  January  2011     Monday           1           0                0
2011-01-04  January  2011    Tuesday           1           0                0
2011-01-05  January  2011  Wednesday           1           0                0

Helper Functions

In [10]: from auto_explore.apis import fetch_fred_data
In [11]: series_list = ['SP500', 'NASDAQCOM', 'DJIA', 'RU2000PR'] # cboe energy sector etf volatility
In [12]: econ_df = fetch_fred_data(series_list)
In [13]: econ_df.head()
Out[13]: 
              SP500  NASDAQCOM      DJIA  RU2000PR
DATE                                              
2011-01-03  1271.87    2691.52  11670.75   1984.61
2011-01-04  1270.20    2681.25  11691.18   1952.99
2011-01-05  1276.56    2702.20  11722.89   1976.01
2011-01-06  1273.85    2709.89  11697.31   1966.88
2011-01-07  1271.50    2703.17  11674.76   1957.96            0

auto_explore.eda

This part of the library was designed to be an interface to automatic exploration.

Simply specify a DataFrame and a list for each of its binary, categorical, numerical and text columns. If applicable, set the target_col as a list with one element (string).

from auto_explore.eda import AutopilotExploratoryAnalysis

args = (df, bin_cols, cat_cols, num_cols, text_cols)
kwargs = dict(target_col=target_col)
ax = AutopilotExploratoryAnalysis(*args, **kwargs)

This object makes available 17 methods for use on the df supplied as an arg. Check out the most recent code on Github.

Other Modules

The actual functionality of the AutopilotExploratoryAnalysis object has been abstracted away and modularized into various files within the auto_explore package. This allows for re-use of the code even if the automated option is not taken.

• auto_explore.viz - Contains all the visualization functions useful in EDA
• auto_explore.featexp - A copy featexp code with custom changes; possible pull request in the future
• auto_explore.apis - Code that fetches data and machine learning models from sources
• auto_explore.notebooks - Formatting code for inside a Jupyter Notebook REPL environment
• auto_explore.stats - Currently only houses best_theoretical_distribution
• auto_explore.datetime - Houses code pertaining to time-series feature generation
• auto_explore.diligence - Houses code that performs sanity checks of various sorts